Detecting and executing data re-ingestion to improve accuracy in a NLP system

ABSTRACT

In some NLP systems, queries are compared to different data sources stored in a corpus to provide an answer to the query. However, the best data sources for answering the query may not currently be contained within the corpus or the data sources in the corpus may contain stale data that provides an inaccurate answer. When receiving a query, the NLP system may evaluate the query to identify a data source that is likely to contain an answer to the query. If the data source is not currently contained within the corpus, the NLP system may ingest the data source. If the data source is already within the corpus, however, the NLP may determine a time-sensitivity value associated with at least some portion of the query. This value may then be used to determine whether the data source should be re-ingested—e.g., the information contained in the corpus is stale.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 13/796,562, filed Mar. 12, 2013. The aforementioned relatedpatent application is herein incorporated by reference in its entirety.

BACKGROUND

The present invention relates to managing data sources in a corpus, andmore specifically, to identifying new data sources for ingestion intothe corpus or determining if a current data source stored in the corpusis stale.

Natural language processing (NLP) is a field of computer science,artificial intelligence, and linguistics concerned with the interactionsbetween computers and human languages. To interact with humans,natural-language computing systems may use a data store (i.e., a corpus)that is parsed and annotated. For example, the computing system may usethe corpus to identify an answer to a question posed by a human user bycorrelating the question to the annotations in the data store.

Before the NLP computing system is able to interact with a user, thecorpus is populated with different text documents. In addition,annotators may parse the text in the corpus to generate metadata aboutthe text. Using the metadata and the stored text, the NLP computingsystem can interact with the user to, for example, answer a posedquestion, diagnosis an illness based on provided symptoms, evaluatefinancial investments, and the like. In a sense, the corpus acts likethe “brain” of the natural-language computing system.

SUMMARY

Embodiments described herein include a system and a computer programproduct that receiving a query for processing by a NLP system andidentify a data source related to the query by associating one or moreelements of the query to the data source. Upon determining that therelated data source is not in a corpus of the NLP system, the system andcomputer program product ingest the related data source into the corpus.Upon determining that the related data source is in the corpus of theNLP system, the system and computer program product determine atime-sensitivity value associated with the query indicating a degree towhich an accurate answer to the query is dependent on a staleness of therelated data source. Upon determining that the time-sensitivity valuesatisfies a staleness threshold, the system and computer program productre-ingest the related data source into the corpus.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments of the invention, briefly summarized above, may be had byreference to the appended drawings.

FIG. 1 is a flow chart for ingesting documents from a data source into acorpus of a NLP system, according to one embodiment described herein.

FIG. 2 is a flow chart for identifying a data source for ingestion orre-ingestion into the corpus, according to one embodiment describedherein.

FIG. 3 is a flow chart for identifying a data source by characterizingelements in a received query, according to one embodiment describedherein.

FIG. 4 is a flow chart for assigning a time-sensitivity value to a queryto determine whether to re-ingest a data source, according to oneembodiment described herein.

FIG. 5 is a flow chart for determining when to re-ingest a data sourceto provide supplemental answers to the received query, according to oneembodiment described herein.

FIG. 6 is a system diagram of a NLP processing system, according to oneembodiment described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements disclosed in oneembodiment may be beneficially utilized on other embodiments withoutspecific recitation.

DETAILED DESCRIPTION

A data store for a natural language processing (NLP) system may includeinformation that originates from a plurality of different datasources—e.g., journals, websites, magazines, reference books, textbooks,and the like. In one embodiment, the information or text from the datasources are converted into a single, shared format and stored as objectsin a data store (i.e., a corpus). For example, an article in a journalmay be formatted differently than an entry in an encyclopedia. Moreover,different journals may have different formats for printing theirrespective articles. Thus, in order to ingest the different documentswith their respective formats, a NLP system may preprocess the documentsto change the different formats into a normalized format (also referredto herein as a “common format”). As used herein, a data source's formatincludes the manner in which the text is arranged. The format mayinclude different formatting elements such as section headers, paragraphheaders, elements in a mark-up language (e.g., HTML and XML tags), andthe like. Additionally, the format used by a data source may specify aparticular hierarchy or order of the formatting elements—e.g., anintroduction section followed by a general discussion section followedby a conclusion section. This process of adding to data source to thecorpus is referred to generally herein as ingestion.

Once the data sources are ingested, a received query may be annotatedand compared to the data stored in the corpus. Based on this comparison,the NLP system may identify one or more answers to the query in thecorpus. In some instances, however, a data source that contains ananswer to the query may not be contained within the corpus, or thecorpus may contain stale data that provides an inaccurate answer. Whenreceiving a query, the NLP system may evaluate the query to identify adata source that may contain an answer to the query. If the data sourceis not currently contained within the corpus, the NLP system may ingestthe data source. If the data source has already been ingested into thecorpus, the NLP may determine a time-sensitivity value associated withat least some portion of the query. This value may then be used todetermine whether the data source should be re-ingested—e.g., theinformation contained in the corpus may be stale.

In another embodiment, after the NLP system compares a query to thecorpus to identify an answer to the query, the NLP system may attempt toidentify one or more supplemental data source that may contain a moreaccurate answer. To do so, the NLP system may identify one or moreconcepts associated with the different elements in the query and usethese concepts to filter through the different data sources stored inthe corpus. In this manner, the NLP system can identify data sourcesthat are related to the query and may, if updated, contain asupplemental answer to the query. The NLP system may then determine ifnew data has recently been added to the data source. If so, the NLPsystem may re-ingest the data source and again compare the query to thecorpus to determine if the re-ingested data source contains an answerthat may be better than the answers that were identified previously. Inthis manner, the NLP system may augment the answers found originally byre-ingesting data sources into the corpus and re-evaluating the queryagainst the updated corpus.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentinvention, a user may access applications (e.g., the data store) orrelated data available in the cloud. For example, the preprocessor thatpopulates the data store could execute on a computing system in thecloud and receive the particular text documents. In such a case, the usecould transmit the text documents to the preprocessor which thengenerates the data store at storage location in the cloud. Doing soallows a user to access this information from any computing systemattached to a network connected to the cloud (e.g., the Internet).

FIG. 1 is a flow chart 100 for generating a corpus for a NLP system,according to one embodiment described herein. The flow chart receivestext documents (e.g., electronic files, portable document format (PDF)files, flat text files, HTML text, and the like) from a plurality ofdifferent data sources 105A-105D which may arrange the text according torespective formats. For example, the webpage 105A may representdifferent web pages that are retrieved from a particular domain, e.g.,Wikipedia® (a registered trademark of the Wikimedia Foundation). Thedomain may include multiple webpages that are each directed to aspecific topic. Although not shown in FIG. 1, a webcrawler may discovereach webpage 105A and transmit these pages to the preprocessor system110 as respective text documents. The domain associated with the webpage105A may define a particular format that is shared by all the webpages105A of that domain. For example, the webpage 105A may include titletags (<title>TITLE</title) followed by body tags (<body>), header tags(<h1> . . . <hN>), and the like. Thus, the data sources 105 may have apredictable format that may be used to parse and identify the relevanttext.

The other data sources shown in FIG. 1 may have formats that arrangetext differently relative to the webpages 105A. For example, the journal105B may always use the same format when publishing articles. The formatmay include, for example, the same primary headers or formattingelements—e.g., introduction, analysis, conclusion, andreferences—arranged in a particular order. Additional formats may beidentified for text documents (e.g., articles, pages, sub-pages, ortopics) received from the other data sources. For example, a pluralityof electronic pages from an encyclopedia may be transmitted to thepreprocessing system 110 which uses a parser to a scan the pages toidentify different topics and the text associated with those topics. Forexample, the encyclopedia may use a special font or spacing whenintroducing a new topic. Using these known characteristics of theformat, the preprocessing system 110 may identify portions of the pagethat correspond to each topic.

The preprocessing system 110 receives the various text documents anduses the respective formats to identify relevant text. In oneembodiment, the preprocessing system 110 may include respective codeportions referred to herein as extension classes that correspond to eachof the formats. A more detailed explanation of extension classes isdescribed in U.S. patent application Ser. No. entitled “ELECTRONICDOCUMENT SOURCE INGESTION FOR NATURAL LANGUAGE PROCESSING SYSTEMS” whichwas filed on Dec. 10, 2012 and is hereby incorporated by reference inits entirety. Using the extension class, the preprocessing system 110identifies the relevant text and associates the text with a formattingelement in a normalized (or common) format. The preprocessing system 110generates objects 115 (e.g., NLP objects) that are commonly formatted.That is, regardless of the particular format used to arrange thereceived text, in the NLP object 115, the text may be arranged based ona shared format between all the objects 115. For example, thepreprocessing system 110 may generate a new object 115 for each topicreceived from a data source (e.g., an article about breast cancer or awebpage about treating prostate cancer). If, however, there is alreadyan object 115 assigned to that topic, the preprocessing system may storethe new text into the already existing object 115. The preprocessingsystem 110 arranges the text in these documents into the normalizedformat where each document is categorized in the object 115. In oneembodiment, the object 115 may be an individual source common analysissystem (CAS) data structure as defined by the Unstructured InformationManagement Architecture (UIMA), an OASIS standard. Once the textdocuments are ingested (i.e., the text is arranged into the objects115), the preprocessing system 110 may place the object 115 or CAS intothe corpus 120.

Selectively Ingesting or Re-Ingesting Data Sources into a Corpus

FIG. 2 is a flow chart for identifying a data source for ingestion orre-ingestion into the corpus, according to one embodiment describedherein. Generally, the ingestion process refers to the technique usedfor adding one or more documents, files, web-pages, and the likeassociated with a data source to the corpus of a NLP system. FIG. 1illustrates one such example for ingesting documents. The terms“ingesting” and “re-ingesting” may be used to distinguish between datasources that are added to the corpus for the first time versus datasources that have been previously ingested. That is, “ingesting a datasource” may refer to the first time documents from a particular datasource have been added to the corpus while “re-ingesting a data source”refers to again performing the ingestion process for documentsassociated with the data source. Re-ingesting a data source may includeingesting new documents that were added to the data source or updatingdocuments that were previously added to the corpus but have sincechanged (or both).

At block 205, a NLP system may receive a query. The query may be phrasedas a question (though this is not necessary) and based on a particularlanguage for human communication—e.g., English, French, Russian, etc. Inone embodiment, the query may be submitted by a user either directly orindirectly. For example, the user may type the query onto a personalcomputer which then sends the query to a NLP system for processing, orthe user may communicate directly with the NLP system, e.g., the systemmay include a voice-recognition application that captures a user'sspoken query. Based on the different elements in the query (i.e., thewords or phrases in the query), the NLP system may search the corpus toidentify an answer to the query.

At block 210, the NLP system identifies a data source which may containan answer to the query by evaluating the elements within the query. Forexample, the NLP system may parse the query and identify data sourcesbased on the elements within the query or from deriving concepts basedon these elements. For example, the query may ask “what is price ofIBM's BladeCenter HX5?” Here, a data source can be directly derived fromthe elements in the query—i.e., a website or pricing database associatedIBM® products (IBM is a registered of International Business Machines inthe United Stated and other jurisdictions). In other embodiments, arelevant data source (or sources) may be identified by indirectlyderiving information from the query such as a general topic associatedwith the query. A more detailed explanation of this block may be foundin the discussion accompanying FIG. 3.

At block 215, the NLP system may determine whether the data sourceidentified at block 210 is currently located in the corpus—i.e., whetherthe data source has been previously ingested. If not, at block 220, theNLP system may ingest the data source. If the data source has beeningested, the NLP system may perform further analysis to determinewhether the data within the corpus is stale. At block 225, the NLPsystem may determine a time-sensitivity value associated with the query.In one embodiment, the query may specify a time or duration—e.g., “Whatis the temperature in Rochester, Minn. yesterday?” Because the word“yesterday” indicates that the answer needs to come from a data sourcethat was ingested since yesterday, the NLP system may set a hightime-sensitivity value. The high time-sensitivity value may indicate,for example, that the staleness of the data source used to provide theanswer may have a large impact on the accurateness of the answer. Incontrast, if the query is whether an actor won a certain award in 2010,the NLP system may set a low time-sensitivity value to the query sincethe data source identified at block 210 needs only to have been updatedsince 2010 to have this answer. A more detailed explanation of block 225is provided in the discussion accompanying FIG. 4.

At block 230, the NLP system may determine whether to re-ingest theidentified data source based on the time-sensitivity value. For example,the NLP system may re-ingest the data source if the time-sensitivityvalue satisfies a predefined threshold. In other embodiments, the NLPsystem may include other factors, such as an estimated time required tore-ingest the data source, when deciding whether to re-ingest the datasource. For example, even if the time-sensitivity value is high, the NLPsystem may not re-ingest the data source if the estimated time is twentyminutes. Alternatively, if the time-sensitivity value is low, the NLPsystem may still re-ingest the data source if the estimated time is onlya few seconds (e.g., the NLP system only has to ingest a smalldocument). Other factors that may be considered before re-ingesting adata source include a time-importance associated with the query (i.e.,whether the requesting entity needs an answer immediately or if theanswer can be delayed to allow for re-ingestion), whether the NLP systemhas already identified a reliable answer from a different data source,or the ability of the NLP system to perform the re-ingestion based onthe current utilization of the system's hardware resources. In oneembodiment, the factors discussed above may also be used by the NLPsystem to determine whether to ingest a new identified data source atblock 220. For example, if ingesting the new data source is estimated totake more than a minute, the NLP system may choose not to ingest thedata source. Instead, the NLP system may rely only on the data sourcesalready ingested into the corpus to provide an answer to the query.

In one embodiment, the NLP system may provide a user with variousoptions before ingesting a new data source at block 220 or re-ingestingan old data source at block 230. In the situation where the NLP systemwants to ingest a new data source, the system may ask the user forpermission to ingest the source. Doing so provides the user with anopportunity to answer the query with only existing sources rather thanwaiting for a new source to be ingested. In this scenario, the NLPsystem may display the estimated time for ingesting the source and letthe user select whether to ingest the source. In addition, the NLPsystem may evaluate the query, determine an answer, and display thisanswer to the user before asking the user whether she would like toingest a new source that may have a more accurate answer. Continuing theexample provided above, the NLP system may use data source already inthe corpus (e.g., a website for selling secondhand electronics) toprovide the price of a used BladeCenter HX5. When displaying this price,the NLP system may state that a more accurate price may be found ifIBM's website (e.g., the data source identified at step 210) is ingestedinto the corpus. In another embodiment, the NLP system may provide aHTML link to the data source where the user can attempt to answer thequery herself. Moreover, the options discussed above may also beprovided to the user in the case where the NLP system is unable to findany answer to the query.

In the situation where the NLP system wants to re-ingest a currentlystored data source, the system may display an estimated time foringesting the source and let the user decided whether to ingest a datasource that may have been updated which provide a more accurate answer.Moreover, the NLP system may have previously processed the query usingthe un-updated data source in the corpus. If an answer was found, theNLP system may display the answer to the user but state that the NLPsystem has determined that the answer may be stale. For example, if thequery asks for the most recent score for a sports team, the NLP systemmay display the answer based on searching an ingested sports website.However, because the query asks for the most recent score, the NLPsystem may display the most recent score found along with the date, butask the user whether she would like to wait for the NLP system tore-ingest the data source and re-process the query based on the updatedcorpus to ensure the answer is accurate.

In one embodiment, method 200 may be performed before the NLP systemprocesses the query. That is, method 200 may be a pre-processingtechnique used for determining whether the corpus should be updatedbefore searching the corpus to identify an answer to the query.Alternatively, method 200 may be performed after or in parallel withsearching the corpus to identify an answer to the query. In one example,the NLP system may perform at least a portion of method 200 after thesystem has already determined an answer for to the query. The NLP maydetermine based on some confidence parameter, however, that the answermay be inaccurate. In response, the NLP system may perform method 200 todetermine if the corpus should be updated by ingesting a new data sourceor re-ingesting a current data source (or both) to potentially find amore accurate answer to the question when the query is reprocessed.

FIG. 3 is a flow chart for identifying a data source by characterizingelements in a received query, according to one embodiment describedherein. Specifically, method 300 is a more detailed explanation of atechnique that the NLP system may perform during block 210 of FIG. 2. Atblock 305, the NLP system may parse the elements of a query to determinewhether the query specifically mentions a data source. An element of thequery may be a single word or a plurality of related words (e.g., aphrase). The embodiments discussed herein may use any NLP technique fordividing words of a query into different elements.

The NLP system may evaluate each of the elements to see if the elementscorrespond to a particular data source. In one embodiment, the NLPsystem may search a specific database (e.g., a list of companiesregistered in a country) or the Internet to determine if one of theelements is a data source. For example, if the query contains the wordIBM, the NLP system may do a web search to identify one or more datasources associated with IBM—e.g., IBM's website, press releases aboutIBM, publications by IBM, etc. The NLP may flag these different datasources for ingestion into the corpus. In this manner, the elements inthe query may be parsed to directly derive different potential datasources that may contain an answer to the query.

At block 310, the NLP system may characterize each element in the queryaccording to a topic associated with the respective elements. In oneembodiment, the NLP system may annotate each of the elements in a query.As part of this process, the NLP system may assign the element toparticular topic or provide a generic description of the element. Forexample, a period of time (e.g., two hours) may be annotated as a timeor a duration. A person's name may be annotated by their job orprofession (e.g., an actor or politician). These annotations aremetadata or derived data that elaborates on the underlying element inthe query. Based on this metadata or characterization of the elements inthe query, the NLP system may indirectly identify a data source relatedto the query.

At block 315, the NLP system may use the characterization of theindividual elements to search for related data source. In oneembodiment, the NLP system may maintain a list of possible data sourcesthat have not been ingested into the corpus. For example, the NLP systemmay have determined previously that a particular data source is notrelevant enough to the types of queries the NLP system typically answersto warrant ingesting the data source into the corpus. Nonetheless, theNLP system may add the data source into a table along with a briefdescription of the source—e.g., its title or an abstract describing thesource. The NLP system may search the table based on thecharacterization of an element to see if there is a match in the table.For example, the NLP system may decide not to ingest a data source whichincludes biographies for famous actors, but instead add a description ofthe data source into the aforementioned table. If the NLP system laterreceives a query where one of the elements is characterized as an actor,the NLP system may search the table and flag the data source containingthe biographies as a potential source for ingestion. Additionally, theNLP system may search the current data sources in the corpus to see ifone of these sources matches or relates to the characterizations of oneof the elements. If so, the NLP system may flag the data source forfurther evaluation (i.e., to determine whether the data source should bere-ingested).

In another embodiment, the NLP system may search the Internet using thecharacterizations to identify a new data source not in the corpus. Forexample, the NLP system may input the characterizations of the elements(e.g., all the different topics associated with the elements) into asearch engine and evaluate the results. The NLP system may generate asummary of the different websites identified by the search engine andcompare the summaries to the characterizations of the elements. If thecharacterizations and the summaries are similar, the NLP system may thenflag the domain of the website as a potential new data source for thecorpus.

Although method 300 was discussed in the context of identifying a singledata source, the NLP system may use method 300 to identify a pluralityof data source which may be new data sources or data sources previouslyingested into the corpus. After identifying one or more candidate datasources, method 300 may return to block 215 of FIG. 2 to determine whataction is appropriate.

As shown, method 300 illustrates performing block 310 (i.e., identifyinga data source indirectly based on a characterization of the elements inthe query) after performing block 305 (i.e., identifying a data sourcedirectly by recognizing a data source in the query); however, in oneembodiment, the NLP system may perform only one of these techniquesrather than performing both as shown in method 300. For example, the NLPsystem may assume that data sources specifically mentioned in the queryare the most relevant and go directly to block 215 of FIG. 2 if thesystem identifies any data sources at block 305. Alternatively, the NLPsystem may first attempt, based on a characterization of the elements,to identify new data sources. If unsuccessful, the system may thendetermine if the query specifically includes a data source.

FIG. 4 is a flow chart for assigning a time-sensitivity value to a queryfor determining whether to re-ingest a data source, according to oneembodiment described herein. Specifically, method 400 is a more detailedexplanation of a technique that the NLP system may perform during block225 of FIG. 2. Accordingly, method 400 may begin after determining thata data source identified at block 210 of FIG. 2 is already in thecorpus. At block 405, the NLP system performs a concept mapping for thedifferent elements in the query. In one embodiment, the concept mappingmay be similar (or the same) as the characterization performed at block310 of FIG. 3. Much like annotating the elements in the query to providea description of the elements, the NLP system may assign a concept toeach element. In one embodiment, the NLP system may identify a lexicalanswer type which is a word or noun phrase in the query that predictsthe type of the answer. One way of filtering potential answers to aquery is to see if the potential answers are the same type as thelexical answer type. For example, if a received query is “how much is a40 inch television,” the NLP may identify the lexical answer type ofthis query as a price. Accordingly, any potential answer should also beassociated with price. Based on this NLP technique or any other suitabletechnique, the NLP system may assign a concept to each element in thequery or to the query as a whole.

At block 410, the NLP system may determine whether the assigned conceptor concepts are associated with a duration or time frame. The query may,for example, stipulate a length of time (e.g., six months, two days,five hours, “how long”, etc.) or a time frame (1942, last week, Dec. 25,2012, current/present, yesterday, etc.) which may be mapped to a conceptrelated to time or duration. At block 415, the NLP system assigns atime-sensitivity value based on the time-related data expressed in thequery. For example, the NLP system may assign a high time-sensitivityvalue if the time-related data happened recently. To do this, in oneembodiment, the NLP system may associate the time-related data in thequery to a fixed reference point—e.g., the current date. A query thatasks “how long was Neil Armstrong on the moon” may have a lowertime-sensitivity value compared to a query that asks “how long wasyesterday's power outage”. Even though these two queries contain thesame time-related data—e.g., the phrase “how long”—the term “yesterday”in the latter query indicates that this query is much more dependent onwhen the underlying data sources have been updated than the former queryrelated to Neil Armstrong's moon landing. Thus, by fixing thetime-related data in the queries to the current date, the NLP system mayassign a customized time-sensitivity value to one or more elements inthe query or the query as a whole.

If the concept mapping does not result in directly identifyingtime-related data, at block 420, the NLP system may determine if theconcepts associated with the elements may be dependent on time. Stateddifferently, the ability to accurately answer queries associated withcertain concepts may depend on the staleness of the data sourcescontaining the answers. For example, the query asking “who is the topranked college basketball team” does not directly include time-relateddata. However, the concept of “rank” is dependent on time since, duringbasketball season, the number one ranked team may change on a weeklybasis. Accordingly, the NLP system may infer from the concepts theappropriate time-sensitivity value. For example, the NLP system mayinclude a predefined table that lists concepts that are dependent ontime (e.g., rankings, stock quotes, movie theater screenings, prices,etc.). If the elements in the query are associated with any of theseconcepts, than the NLP system may assign a higher time-sensitivityvalue.

In other embodiments, the NLP system may combine (or considerindependently) the verb tense and the concepts to determine atime-sensitivity value. In the previous example, the verb “is” may alsobe used to influence the time-sensitivity score. On the other hand, ifthe words in the query use a past-tense structure, the NLP system maydecrease the time-sensitivity score. In addition to the verb tense,other factors may be considered to set the time-sensitivity value. Forexample, if the query includes a historical event, when that eventoccurred may be used to assign the time-sensitivity value. Thus, the NLPsystem may combine a variety of different factors or considerations withconcept mapping in order to generate a time-sensitivity value for thequery. Once the time-sensitivity value is assigned, the NLP system mayproceed to block 230 of FIG. 2 to determine whether to re-ingest theidentified data source based on the time-sensitivity value. For example,the NLP system may compare the time-sensitivity value to a thresholdvalue or compare the value to a timestamp in the corpus indicating thelast time the data from the data source was updated.

If, however, the NLP system does not identify any time dependentconcepts in the query, method 300 proceeds to block 425 where the NLPsystem may answer the query without re-ingesting the identified datasource. In one embodiment, the NLP system may, when answering the queryusing the identified data source, inform the user the last time the datasource was updated (e.g., a timestamp). The user may then independentlydetermine that the data source is stale and request that the NLP systemre-ingest the data source and re-process the query.

Augmenting Candidate Answers by Re-Ingesting a Data Source

FIG. 5 is a flow chart for determining when to re-ingest a data sourceto provide supplemental answers to the received query, according to oneembodiment described herein. At block 505, the NLP system receives aquery from, for example, a human user. At block 510, the NLP systemperforms concept mapping for the different elements in the query or forthe query as a whole. In one embodiment, the concept mapping performedat block 510 of method 500 may be the same or similar to the conceptmapping performed at block 405 of FIG. 4. Alternatively, method 500 maycharacterize the different elements to identify a topic associated withthe elements as discussed at block 310 of FIG. 3.

At block 515, the NLP system determines one or more answers to thequery. That is, using any NLP technique, the system may compare theelements in the query to the corpus to identify potential answers to thequery. In one embodiment, the NLP system may also associate confidencescores to each of the answers which indicate the trustworthiness of theanswers. The NLP system may determine based on the confidence scoreswhether to augment the answer by determining whether the one or moredata sources that provide the answer should be re-ingested. For example,the NLP system may include a predefined threshold value that is comparedto the answers. If the answers' confidence scores do not satisfy orexceed the threshold, the NLP system may determine to augment theanswers by determining if the data sources that provided the answers arestale. That is, the confidence scores may be improved if the datasources are re-ingested to include updated data.

In one embodiment, method 500 may be performed even if no answers werefound when comparing the query to the corpus. That is, because none ofthe data sources in the corpus currently contain an answer, the NLPsystem may attempt to determine one or more data sources to re-ingest.Further still, method 500 may be performed before determining an answerto the query. Stated more generally, the NLP system may perform method500 independent of the results of comparing the query to the corpus. Inthis situation, the NLP system may, each time a query is received, usethe concept mapping done at block 510 to identify data sources relatedto the query that are stale and may need to be re-ingested.

At block 520, the NLP system may filter the data sources in the corpusbased on the concepts identified in the query. That is, in addition todetermining whether to re-ingest data sources that have already providedan answer at block 515, the NLP system may determine whether other datasources which did not provide an answer, but are nonetheless related tothe query, should be considered. Assume that a query asks what dosage ofa particular drug is appropriate for treating breast cancer. The NLPsystem may identify certain concepts based on the query such as“dosage”, “treating”, “breast cancer”, “cancer”, etc. When ingesting adata source, the NLP system may generate a summary of the data source orkey concepts of the data source. Accordingly, the NLP system maycross-reference the concepts identified in the query with the summariesof the sources in the corpus. When doing so, the NLP system may identifya data source that is generally related to drugs used for treatingbreast cancer. On the other hand, sources in the corpus that do notinclude these concepts may be de-emphasized—e.g., sources related toallergies or sports-injuries may be filtered out. In this manner, theNLP system may identify one or more supplemental data sources thatcurrently do not answer the query but are expected to have an answer tothe query based on the similarities between the concepts in the queryand a general description of the supplemental data source.

At block 525, the NLP system determines whether the supplemental datasource has been updated. For example, the supplemental data source maybe a journal that has published another article or released a new issuesince the being ingested. In one embodiment, the NLP system may comparea date associated with the data source to a timestamp in the corpusindicating when the data source was last ingested. To identify a dateassociated with the data source, the NLP system may use a web-crawler toidentify when the web pages in a domain have been updated, query adatabase to retrieve timestamps for the data stored in the database,parse an electronically digitized document (e.g., a PDF of a journal),and the like.

If the NLP system determines that the date associated with the datasource that is more recent than the timestamp, the system may, at block530, re-ingest the data source. In one embodiment, the NLP system mayre-ingest only a portion of the data source (e.g., a new article or themost recent issue) rather than re-ingesting the entire data source(e.g., all the volumes of a data source).

In one embodiment, the NLP system may provide a user with variousoptions before re-ingesting the supplemental data source at block 530.The NLP system may, for example, ask the user for permission tore-ingest the source. The NLP system may display the answers alreadyfound at block 525 as well as statement that the answers may not bereliable (e.g., the confidence scores are low). The NLP system may alsotell the user that a source that is related to the query (e.g., sharessimilar concepts) has recently published new data that could bere-ingested to possible find a better answer. Thus, the user can decidewhether to wait for the NLP system to re-ingest the data source. The NLPsystem may also display the estimated time for ingesting the sourcewhich may influence the user's decision. Alternatively or additionally,the NLP system may provide a HTML link to the data source where the usercan access the updated data source herself. In one embodiment, theseoptions may also be provided to the user in the case where the NLPsystem is unable to find any answer to the query at block 515.

Assuming the supplemental data source is re-ingested, at block 535, theNLP system may evaluate the query using the updated corpus. Inembodiments where the NLP system has already compared the query to thecorpus, at block 535, the NLP system is performing the same processagain but this time the corpus includes updated information from there-ingested data source.

A Computing System for Hosting the NLP System

FIG. 6 is a system diagram of a NLP system 615, according to oneembodiment described herein. As shown, NLP system 615 may be hosted on acomputing system 600. Computing system 600 may include a singlecomputing device in a single chassis (e.g., a server) or a plurality ofcomputing devices that are interconnected. For example, computing system600 may include multiple servers in a data center coupled to astorage-area network which are used when executing the NLP system 615.Accordingly, processor 605 and memory 610 may include multipleindividual processors or memory elements that are distributed across theinterconnected computing devices. Processor 605 represents any number ofprocessor units that are capable of performing the functions describedherein. Moreover, the processor units may be single core or multi-coreunits. Memory 610 may include non-volatile as well as volatile memoryelements and may be shared by multiple computing devices in computingsystem 600.

As shown, memory 610 stores NLP system 615 which includes apreprocessing system 110, a query processing system 620, and a corpus120. The description of the preprocessing system 110 and corpus 120provided in FIG. 1 may also be applied here. The query processing system620 may be tasked with comparing a query received by the NLP system 615to corpus 120 in order to provide one or more answers to the query. Theembodiments described herein are not limited to any particular NLPtechnique for implementing the query processing system 620 and may beapplied to any technique that relies on a corpus to answer queries.

CONCLUSION

In some NLP systems, queries are compared to different data sourcesstored in a corpus to provide an answer to the query. However, the bestdata sources for answering the query may not currently be containedwithin the corpus or the data sources in the corpus may contain staledata that provides an inaccurate answer. When receiving a query, the NLPsystem may evaluate the query to identify a data source that is likelyto contain an answer to the query. If the data source is not currentlycontained within the corpus, the NLP system may ingest the data source.If the data source is already within the corpus, however, the NLP maydetermine a time-sensitivity value associated with at least some portionof the query. This value may then be used to determine whether the datasource should be re-ingested—e.g., the information contained in thecorpus is stale.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method comprising: receiving first and secondqueries for processing by a natural language processing (NLP) system;identifying first and second data sources related to the first andsecond queries, respectively, by associating one or more elements of thefirst and second queries to the first and second data sources; upondetermining that the first related data source is not in a corpus of theNLP system, ingesting the first related data source into the corpus,wherein the corpus includes a data store comprising information from oneor more data sources formatted and stored into one or more objects; andupon determining that the second related data source is in the corpus ofthe NLP system: before searching the second data source for at least oneanswer to the second query, determining a time-sensitivity valueassociated with the second query indicating a degree to which anaccurate answer to the second query is dependent on a staleness of thesecond related data source, and upon determining that thetime-sensitivity value satisfies a staleness threshold, re-ingesting thesecond related data source into the corpus.
 2. The method of claim 1,further comprising: performing a NLP technique to compare the first andsecond queries to the corpus to identify respective answers to the firstand second queries after one of (i) ingesting the first related datasource into the corpus and (ii) re-ingesting the second related datasource into the corpus; and transmitting the respective answers to atleast one entity submitting the first and second queries.
 3. The methodof claim 1, wherein identifying the first and second related datasources further comprises: parsing the first and second queries toidentify respective elements of the first and second queries thatspecifically provides the first and second data sources; characterizingthe elements in the first and second queries by assigning respectivetopics to the elements; and identifying the first and second relateddata sources by comparing the topics to a plurality of topics associatedwith different data sources.
 4. The method of claim 1, whereindetermining the time-sensitivity value associated with the second queryfurther comprises: performing a concept mapping to assign a concept toat least one element in the query; upon determining that the concept isrelated to duration or time, assigning the time-sensitivity value basedon duration or time specified in the concept; and upon determining thatthe concept is dependent on time by matching the concept to apre-defined list of time-dependent concepts, assigning thetime-sensitivity value based on the time dependency of the concept. 5.The method of claim 1, wherein determining that the time-sensitivityvalue satisfies the staleness threshold, further comprises determiningwhether the second related data source is stale by comparing atime-stamp associated with the second related data source to thetime-sensitivity value.
 6. The method of claim 1, further comprising,before ingesting or re-ingesting the first and second data sources,receiving permission from a user submitting the first and second queriesto ingest or re-ingest the data sources into the corpus.
 7. The methodof claim 1, wherein data in the one or more data sources is organizedbased on a common format in the corpus.